{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 新闻言论自动提取\n",
    "步骤：\n",
    "1. 构建word2vec词向量模型(可用维基百科数据)利用搜索方法找出和说等相关的词，然后进行筛选，保存到文件备用\n",
    "2. 把步骤1的词向量模型对象的向量数据保存(model.wv类保存成.kv文件)，后续计算句子相似度用，备用\n",
    "3. 利用pyltp等库对输入新闻进行解析，获取详细依存关系等\n",
    "4. 利用3种获得数据进行分析，找出说话的人和他说的话\n",
    "5. 如果4中找出了某人说的话，用该句话和后几个句子(实际调整)进行句子向量化分析，判断后面句子和该句话的相似度，当大于某个阈值时，认为后面句子也是说话的内容。以此来判断说话截止\n",
    "6. 搭建相应前后端框架，最后利用web框架展示成果"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true
   },
   "source": [
    "## 步骤1-2实现代码\n",
    "word2vec模型构建在这不做讲解，仅实现下如何搜索相关词"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "code_folding": [],
    "hidden": true
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "D:\\Design Software\\Anaconda\\envs\\py36\\lib\\site-packages\\gensim\\utils.py:1197: UserWarning: detected Windows; aliasing chunkize to chunkize_serial\n",
      "  warnings.warn(\"detected Windows; aliasing chunkize to chunkize_serial\")\n"
     ]
    }
   ],
   "source": [
    "from gensim.models import KeyedVectors\n",
    "from collections import defaultdict\n",
    "import time\n",
    "def get_related_words_im(initial_words, model):\n",
    "    \"\"\"\n",
    "    @initial_words are initial words we already know. is a list input\n",
    "    @model is the word2vec model, better be model vectors , more fast for search\n",
    "    \"\"\"\n",
    "    unseen = initial_words\n",
    "    seen = defaultdict(int)\n",
    "    \n",
    "    max_size = 200  # could be greater\n",
    " \n",
    "    current_layer = 0\n",
    "    current_layer_length = len(unseen)\n",
    "    current_length = len(initial_words)  \n",
    "    \n",
    "    total_most_similar_find_time = 0\n",
    "    cache_dict = defaultdict(list)  #添加缓存，防止重复调用most_similar，因为调用most_similar费时间\n",
    "    while unseen and len(seen) < max_size:\n",
    "        if len(seen) % 50 == 0: \n",
    "            print('seen length : {}'.format(len(seen)))\n",
    "            \n",
    "        #print(unseen)          \n",
    "        node = unseen.pop(0)#类似广度搜索\n",
    "        \n",
    "        current_length -= 1        \n",
    "        if node in cache_dict:\n",
    "            new_expanding = cache_dict[node]\n",
    "        else:\n",
    "            time_point = time.time()\n",
    "            new_expanding = [w for w, s in model.most_similar(node, topn=20)]\n",
    "            cache_dict[node] = new_expanding\n",
    "            total_most_similar_find_time += time.time()- time_point \n",
    "          \n",
    "        unseen += new_expanding      \n",
    "        #当前层数遍历完，说明此时的unseen里保存的是下一层所有结点,更新层数和下一层结点个数\n",
    "        if current_length ==0: \n",
    "            current_layer += 1\n",
    "            current_layer_length = len(unseen)\n",
    "            current_length  = len(unseen)    \n",
    "        \n",
    "        #if seen['说']== 2:\n",
    "        #    break    \n",
    "        # node 节点的权值，可以在这里考虑优化搜索效果\n",
    "        seen[node] +=  (1 - current_layer*0.1) * ((current_length/current_layer_length)*0.2 + 0.8)\n",
    "        \n",
    "        # optimal: 1. score function could be revised\n",
    "        # optimal: 2. using dymanic programming to reduce computing time\n",
    "    total_most_similar_find_time = int(total_most_similar_find_time)\n",
    "    print(current_layer)\n",
    "    print('total_most_similar_find_time:{}min {}s'.\\\n",
    "          format(total_most_similar_find_time//60 ,total_most_similar_find_time%60))\n",
    "    return seen"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "D:\\Design Software\\Anaconda\\envs\\py36\\lib\\site-packages\\smart_open\\smart_open_lib.py:398: UserWarning: This function is deprecated, use smart_open.open instead. See the migration notes for details: https://github.com/RaRe-Technologies/smart_open/blob/master/README.rst#migrating-to-the-new-open-function\n",
      "  'See the migration notes for details: %s' % _MIGRATION_NOTES_URL\n"
     ]
    }
   ],
   "source": [
    "fname = r\"E:/MYGIT/model/wiki_stopwords/wiki_word2vec.kv\"\n",
    "model_wv = KeyedVectors.load(fname, mmap='r')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "seen length : 0\n",
      "seen length : 50\n",
      "seen length : 100\n",
      "seen length : 150\n",
      "seen length : 150\n",
      "seen length : 150\n",
      "2\n",
      "total_most_similar_find_time:2min 10s\n",
      "Wall time: 2min 10s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "related_words  = get_related_words_im(['说', '表示'], model_wv)\n",
    "result_sorted = sorted(related_words.items(), key=lambda x: x[1], reverse=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('说', 7.1913),\n",
       " ('认为', 4.8077000000000005),\n",
       " ('表示', 4.1393),\n",
       " ('指出', 4.0452),\n",
       " ('称', 4.0292)]"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "result_sorted[:5]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "想要本文提取中获得更好的效果，我们需要人为地在以上搜索结果中进行筛选剔除(总会有一些词不是我们想要的)。然后把结果保存至文件备用"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 步骤3-4实现代码"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "code_folding": [
     13,
     39,
     48,
     55,
     66,
     74,
     81,
     109,
     115,
     156,
     167,
     185,
     196
    ]
   },
   "outputs": [],
   "source": [
    "from collections import defaultdict\n",
    "from pyltp import Segmentor\n",
    "from pyltp import Postagger\n",
    "from pyltp import NamedEntityRecognizer\n",
    "from pyltp import SentenceSplitter\n",
    "from pyltp import Parser\n",
    "from gensim.models import KeyedVectors\n",
    "import numpy as np\n",
    "import jieba\n",
    "import os\n",
    "import re\n",
    "import time\n",
    "from collections import Counter\n",
    "\n",
    "def get_sentences_vec(model_wv, sent_list, word_frequence):\n",
    "    # 句子向量化处理\n",
    "    a = 0.001\n",
    "    row = model_wv.vector_size\n",
    "    col = len(sent_list)\n",
    "    sent_mat = np.zeros((row, col))\n",
    "    for i, sent in enumerate(sent_list):\n",
    "        length = len(sent)\n",
    "        sent_vec = np.zeros(row)\n",
    "        for word in sent:\n",
    "            pw = word_frequence[word]\n",
    "            w = a / (a + pw)\n",
    "            try:\n",
    "                vec = np.array(model_wv[word])\n",
    "                sent_vec += w * vec\n",
    "            except:\n",
    "                pass\n",
    "        sent_mat[:, i] += sent_vec\n",
    "        sent_mat[:, i] /= length\n",
    "\n",
    "    # PCA处理\n",
    "    sent_mat = np.mat(sent_mat)\n",
    "    u, s, vh = np.linalg.svd(sent_mat)\n",
    "    sent_mat = sent_mat - u * u.T * sent_mat\n",
    "    return sent_mat\n",
    "\n",
    "def get_word_frequence(words):\n",
    "    #这里不做停用次处理，直接在计算句子向量时候，如果找不到该词，直接跳过\n",
    "    word_list = []\n",
    "    for word in words:\n",
    "        word_list += word\n",
    "    word_frequence = Counter(word_list)\n",
    "    return word_frequence\n",
    "\n",
    "# 计算余弦相似度\n",
    "def cos_similarity(v1, v2):\n",
    "    #输入向量维度不一致\n",
    "    if len(v1) != len(v2):\n",
    "        return 0\n",
    "    return np.vdot(v1,v2) / (np.linalg.norm(v1) * np.linalg.norm(v2))\n",
    "\n",
    "# 返回句子向量矩阵中各列向量与第一列向量的相似度\n",
    "def calcu_similarity(sent_mat):\n",
    "    #采用点积的方法计算\n",
    "    first = np.array(sent_mat[:, 0]).flatten()\n",
    "    col = sent_mat.shape[1]\n",
    "    sims = []\n",
    "    for i in range(1, col):\n",
    "        vec = np.array(sent_mat[:, i]).flatten()\n",
    "        sims.append(cos_similarity(first, vec))\n",
    "    return sims\n",
    "\n",
    "#获取相似度结果\n",
    "def get_similarity_result(word_list_all, model_wv):\n",
    "    word_frequence = get_word_frequence(word_list_all)\n",
    "    sent_mat = get_sentences_vec(model_wv, word_list_all,word_frequence)\n",
    "    sim = calcu_similarity(sent_mat)\n",
    "    return sim\n",
    "\n",
    "\n",
    "#分句\n",
    "def cut_sentence(string):\n",
    "    \"\"\"@string contain many sentence\"\"\"\n",
    "    from pyltp import SentenceSplitter\n",
    "    sents = SentenceSplitter.split(string)  # 分句\n",
    "    return list(sents)\n",
    "\n",
    "#加载模型\n",
    "def load_all_model():\n",
    "    \"\"\"返回分词，词性标注，命名实体识别，依存解析等实例对象\"\"\"\n",
    "    LTP_DATA_DIR = 'E:/MYGIT/Project/ltp_data'  # ltp模型目录的路径\n",
    "    cws_model_path = os.path.join(LTP_DATA_DIR, 'cws.model')  # 分词模型路径，模型名称为`cws.model`\n",
    "    segmentor = Segmentor()  # 初始化实例\n",
    "    segmentor.load_with_lexicon(cws_model_path, '../ltp_data/cut_external_dict/cut_external_dict')  # 加载模型\n",
    "    \n",
    "    LTP_DATA_DIR = 'E:/MYGIT/Project/ltp_data'  # ltp模型目录的路径\n",
    "    pos_model_path = os.path.join(LTP_DATA_DIR, 'pos.model')  # 词性标注模型路径，模型名称为`pos.model`\n",
    "    postagger = Postagger() # 初始化实例\n",
    "    postagger.load_with_lexicon(pos_model_path, '../ltp_data/pos_external_dict/pos_external_dict')  # 加载模型\n",
    "\n",
    "    LTP_DATA_DIR = 'E:/MYGIT/Project/ltp_data'  # ltp模型目录的路径\n",
    "    ner_model_path = os.path.join(LTP_DATA_DIR, 'ner.model')  # 命名实体识别模型路径，模型名称为`pos.model`\n",
    "    recognizer = NamedEntityRecognizer() # 初始化实例\n",
    "    recognizer.load(ner_model_path)  # 加载模型\n",
    "\n",
    "    LTP_DATA_DIR = 'E:/MYGIT/Project/ltp_data'  # ltp模型目录的路径\n",
    "    par_model_path = os.path.join(LTP_DATA_DIR, 'parser.model')  # 依存句法分析模型路径，模型名称为`parser.model`   \n",
    "    parser = Parser() # 初始化实例\n",
    "    parser.load(par_model_path)  # 加载模型\n",
    "    \n",
    "    fname = r\"E:/MYGIT/model/wiki_stopwords/wiki_word2vec.kv\"\n",
    "    #model_wv.save(fname)\n",
    "    model_wv = KeyedVectors.load(fname, mmap='r')\n",
    "    return [segmentor, postagger, recognizer, parser,model_wv]\n",
    "\n",
    "#释放模型\n",
    "def release_all_model(model_list):\n",
    "    del(model_list[-1])\n",
    "    for each in model_list:\n",
    "        each.release()\n",
    "\n",
    "#获取人名或机构名\n",
    "def get_name(word_list, prase, ner_list, pos_list,say_index):\n",
    "    # 合并说话对象名字\n",
    "    index = -1\n",
    "    for arc in prase:\n",
    "        if arc.relation == \"SBV\" and arc.head == say_index+1:\n",
    "            index = prase.index(arc)\n",
    "            break  \n",
    "    #第二种情况\n",
    "    if index == -1:\n",
    "        for arc in prase:\n",
    "            if arc.relation == \"SBV\" and arc.head == prase[say_index].head and prase[say_index].relation ==\"COO\":\n",
    "                index = prase.index(arc)\n",
    "    #两种情况都没找到\n",
    "    if index == -1:return ''\n",
    "    \n",
    "    Entity = ['S-Nh', 'S-Ni', 'B-Nh', 'B-Ni', 'I-Nh', 'I-Ni', 'E-Nh', 'E-Ni']  # 命名实体标记\n",
    "    name = word_list[index]\n",
    "    \n",
    "    if ner_list[index] in Entity: return name\n",
    "    if pos_list[index] not in ['n', 'nh']:return ''\n",
    "    \n",
    "    pre=word_list[:index]#前半部分\n",
    "    pos=word_list[index+1:]#后半部分\n",
    "    while pre:\n",
    "        w = pre.pop(-1)\n",
    "        w_index = word_list.index(w)\n",
    "\n",
    "        if prase[w_index].relation == 'ADV': continue\n",
    "        if prase[w_index].relation in ['WP', 'ATT', 'SVB'] and (w not in ['，','。','、','）','（']):\n",
    "            name = w + name\n",
    "        else:\n",
    "            pre = False          \n",
    "    while pos:\n",
    "        w = pos.pop(0)\n",
    "        w_index = word_list.index(w)\n",
    "        if prase[w_index].relation in ['WP', 'LAD', 'COO', 'RAD'] and w_index < say_index and (w not in ['，', '。', '、', '）', '（']):\n",
    "            name = name + w # 向后拼接\n",
    "        else: #中断拼接直接返回\n",
    "            pos = False\n",
    "    return name   \n",
    " \n",
    "def get_under_node(parent_node, parse, relation):\n",
    "    index = []\n",
    "    for arc in parse:\n",
    "        if arc.relation == relation and arc.head == parent_node +1:\n",
    "            index.append(parse.index(arc))\n",
    "    try:\n",
    "        return index\n",
    "    except NameError:\n",
    "        return -1 \n",
    "    \n",
    "#找出输入所有节点中最小的索引和最大的索引，然后返回两者间的句子\n",
    "def node_under_sentence(parent_point, den_parsing_list, word_list):    \n",
    "    words_index = []\n",
    "    #搜索子节点\n",
    "    if parent_point == [] or parent_point[0] < 0 :return ''\n",
    "    for point in parent_point:\n",
    "        start = [point]\n",
    "        words_index.append(start[0])\n",
    "        while start!= []:\n",
    "            cur = start.pop(0)\n",
    "            for i, arc in enumerate(den_parsing_list):\n",
    "                if arc.head == cur + 1:\n",
    "                    words_index.append(i)\n",
    "                    start.append(i)\n",
    "    words_index = sorted(words_index)\n",
    "    #print(words_index, len(word_list))\n",
    "    return ''.join(word_list[words_index[0]:words_index[-1]+1])\n",
    "\n",
    "#从文件中提取与说相关的词\n",
    "def get_relatedwords():\n",
    "    path  = 'E:/MYGIT/Project/NLP-Project-1/related_words/saywords'\n",
    "    relatedwords_list = []\n",
    "    with open(path, encoding='utf-8') as f:\n",
    "        line_str= f.readline()\n",
    "        while line_str  != '':\n",
    "            relatedwords_list.append(line_str.strip('\\n'))\n",
    "            line_str = f.readline()\n",
    "    relatedwords_list = tuple(relatedwords_list)\n",
    "    return relatedwords_list\n",
    "\n",
    "def find_opinion_of_someone(input_news, say_related, model_list):\n",
    "    # 输入文本进行分句\n",
    "    segmentor = model_list[0]\n",
    "    postagger = model_list[1]\n",
    "    recognizer= model_list[2]\n",
    "    parser= model_list[3]\n",
    "    model_wv = model_list[4]\n",
    "\n",
    "    sentence_list = cut_sentence(input_news)\n",
    "    sentence_list = [sen for sen in sentence_list if len(sen) > 3]\n",
    "    Entity = ['S-Nh', 'S-Ni', 'B-Nh', 'B-Ni', 'I-Nh', 'I-Ni', 'E-Nh', 'E-Ni']  # 命名实体标记\n",
    "    pro_news_dict = defaultdict(list) #用来存储可能是多句的句子\n",
    "    news_dict = defaultdict(list)\n",
    "    \n",
    "    word_list_all = [list(segmentor.segment(sentence)) for sentence in sentence_list]  # 分词\n",
    "    #name_list = [] #用于保存所有人名\n",
    "    for _i, sentence in enumerate(sentence_list):\n",
    "        #print('\\n——————————句子:{} 处理过程——————————'.format(_i + 1))\n",
    "        #time_point_1 = time.time()\n",
    "        word_list = word_list_all[_i]  #     \n",
    "        pos_list = list(postagger.postag(word_list))  # 词性分析\n",
    "        ner_list = list(recognizer.recognize(word_list, pos_list))  # 命名实体提取\n",
    "        den_parsing_list = list(parser.parse(word_list, pos_list))  # 依存关系\n",
    "                        \n",
    "        # 获取命名实体和说相关词,同时获取索引值\n",
    "        pro_say_word = [(a, i) for i, a in enumerate(word_list) if a in say_related]  \n",
    "        if pro_say_word==[]:continue\n",
    "        \n",
    "        # 找到说的主语\n",
    "        for say in pro_say_word:\n",
    "            name = get_name(word_list, den_parsing_list, ner_list, pos_list, say[1])\n",
    "            #print('第{}句子, sayword:{} name:{}'.format(_i, say, name))\n",
    "            if name != '':\n",
    "                say_underword = []\n",
    "                index = get_under_node(say[1], den_parsing_list, 'VOB')           \n",
    "                if index != -1: \n",
    "                    _flag = len(index)\n",
    "                    say_underword += index\n",
    "                    index = get_under_node(say[1], den_parsing_list, 'COO')\n",
    "                    if index != -1: say_underword += index\n",
    "                        \n",
    "                #特殊情况处理1\n",
    "                if len(say_underword) > _flag:\n",
    "                    for _node in range(_flag):\n",
    "                        if pos_list[say_underword[_node]] in ['n','nh']:\n",
    "                            say_underword.pop(0)\n",
    "                \n",
    "                #特殊情况处理2\n",
    "                saying = node_under_sentence(say_underword, den_parsing_list, word_list)\n",
    "                if not saying:\n",
    "                    if (den_parsing_list[say[1]].relation =='POB' and den_parsing_list[den_parsing_list[say[1]].head-1].relation == 'ADV'):\n",
    "                        saying = ''.join(word_list[say[1]+1:])\n",
    "                        saying = saying.strip('，')\n",
    "                    if not saying:\n",
    "                        if _i>0:\n",
    "                            quotations = re.findall(r'“(.+?)”', sentence_list[_i-1])\n",
    "                            if quotations and len(quotations[-1])> 6: saying = quotations[-1]\n",
    "                if saying != '':\n",
    "                    if saying[-5:] in sentence[-9:]:\n",
    "                        words1 = jieba.lcut(saying)\n",
    "                        _word_list = sentence_list[_i+1:_i+4] #切片的话_i+4如果超出则取到末尾\n",
    "                        _word_list.insert(0, words1)\n",
    "                        sim = get_similarity_result(_word_list, model_wv)\n",
    "                        #print(sim)\n",
    "                        for i_sim, _sim in enumerate(sim):\n",
    "                            if _sim > 0.85:\n",
    "                                pro_news_dict[_i] += [sentence_list[_i + i_sim+1]]\n",
    "                            else:\n",
    "                                break\n",
    "                    news_dict[_i] = [name, saying]\n",
    "                    break  \n",
    "    \n",
    "    #对多句添加进行判断，如果下一句里没说，或没提取出来，则拼接到上一句，避免重复\n",
    "    for key in pro_news_dict.keys():\n",
    "        for _i in range(len(pro_news_dict[key])):\n",
    "            if key+_i+1 not in news_dict.keys():\n",
    "                news_dict[key][1] += pro_news_dict[key][_i]\n",
    "            else:\n",
    "                break \n",
    "    return news_dict"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 步骤5原理与实现代码"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![SentenceEembedding](./img/sentence_embedding.png)\n",
    "\n",
    "[句子向量化论文参考](https://openreview.net/pdf?id=SyK00v5xx)\n",
    "- line2：计算每条句子的向量，以列存储 原文是 compute the weighted average of the word vector\n",
    "- line6：原文是 remove the projections of the average vectors on their first singular verctor(common component removal),至于为什么line6这样计算会是common component removal，还没弄明白，有空再深究\n",
    "- 最后得到的矩阵是M×N的 M为词向量维度，N为句子个数，可以看出每个句子向量都是以列保存的"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "code_folding": [
     4,
     30,
     39,
     46
    ]
   },
   "outputs": [],
   "source": [
    "from gensim.models import KeyedVectors\n",
    "import logging\n",
    "\n",
    "#该函数对应的是上诉图片中伪代码的实现过程\n",
    "def get_sentences_vec(model_wv, sent_list, word_frequence):\n",
    "    # 句子向量化处理\n",
    "    a = 0.001\n",
    "    row = model_wv.vector_size\n",
    "    col = len(sent_list)\n",
    "    sent_mat = np.zeros((row, col))\n",
    "    for i, sent in enumerate(sent_list):\n",
    "        length = len(sent)\n",
    "        sent_vec = np.zeros(row)\n",
    "        for word in sent:\n",
    "            pw = word_frequence[word]\n",
    "            w = a / (a + pw)\n",
    "            try:\n",
    "                vec = np.array(model_wv[word])\n",
    "                sent_vec += w * vec\n",
    "            except:\n",
    "                pass\n",
    "        sent_mat[:, i] += sent_vec\n",
    "        sent_mat[:, i] /= length\n",
    "\n",
    "    # PCA处理\n",
    "    sent_mat = np.mat(sent_mat)\n",
    "    u, s, vh = np.linalg.svd(sent_mat)\n",
    "    sent_mat = sent_mat - u * u.T * sent_mat\n",
    "    return sent_mat\n",
    "\n",
    "def get_word_frequence(words):\n",
    "    #这里不做停用次处理，直接在计算句子向量时候，如果找不到该词，直接跳过\n",
    "    word_list = []\n",
    "    for word in words:\n",
    "        word_list += word\n",
    "    word_frequence = Counter(word_list)\n",
    "    return word_frequence\n",
    "\n",
    "# 计算余弦相似度\n",
    "def cos_similarity(v1, v2):\n",
    "    #输入向量维度不一致\n",
    "    if len(v1) != len(v2):\n",
    "        return 0\n",
    "    return np.vdot(v1,v2) / (np.linalg.norm(v1) * np.linalg.norm(v2))\n",
    "\n",
    "# 返回句子向量矩阵中各列向量与第一列向量的相似度\n",
    "def calcu_similarity(sent_mat):\n",
    "    #采用点积的方法计算\n",
    "    first = np.array(sent_mat[:, 0]).flatten()\n",
    "    col = sent_mat.shape[1]\n",
    "    sims = []\n",
    "    for i in range(1, col):\n",
    "        vec = np.array(sent_mat[:, i]).flatten()\n",
    "        sims.append(cos_similarity(first, vec))\n",
    "    return sims\n",
    "\n",
    "#获取相似度结果，相当于封装以上几个函数，简化调用\n",
    "def get_similarity_result(word_list_all, model_wv):\n",
    "    word_frequence = get_word_frequence(word_list_all)\n",
    "    sent_mat = get_sentences_vec(model_wv, word_list_all,word_frequence)\n",
    "    sim = calcu_similarity(sent_mat)\n",
    "    return sim"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 模型效果测试"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "input_news = '''中国国务院新闻办公室当日举行“建设国家公园省 传递大美青海情”发布会。刘宁说，时至今日，青海绝不是仅仅保护一两种动物。现在三江源区有蹄类的动物已经恢复到18万只。青海有603种动物属种，有了大面积恢复，生产力、生长力在不断增强。\n",
    "　　青海有中国最大的世界自然遗产地可可西里，它在4500米的高原上，有6.03万平方公里，是三江源国家公园重要的组成部分。可可西里有一个卓乃湖，是藏羚羊的栖息地、繁育地。刘宁特别提到了为了保护藏羚羊不被非法猎取而牺牲生命的索南达杰。\n",
    "　　“最近我们遇到了难题。”刘宁说，由于生态恢复、植被恢复和降雨量的增加，还有局部气温升高，导致卓乃湖降雨量丰沛，与其他三个湖连起来，一旦漫溢可能冲刷掉下游大量的沼泽、湿地和冻土，甚至对沿途经过的青藏铁路、青藏公路、输电线路等带来影响。今年7月15日青海开始应急工程施工，1800多人和500多台机械在4500米的高原上紧急作业，到8月25日开始放水，现在险情总体可控。\n",
    "　　刘宁通过这个例子说，在顺应自然、尊重自然的同时，保护自然绝不是盲目等待自然的演替，而是在人力可能干预的情况下，对于自然灾害可能造成的破坏，进行积极主动的保护。\n",
    "　　他同时表示，青海和西藏的羌塘、新疆的阿尔金山是联动的。如果出现了对二级保护动物、一级保护动物带来的伤害，都要互相协同追责。\n",
    "　　刘宁说，在青海有一个阿尼玛卿雪山是小冰川时代的遗址，山里有360个湖泊，最高峰是5369米，现在出于保护的原因被关闭了。青海湖还有一个鸟岛，有几千种候鸟，现在也进行了有效的管理。\n",
    "　　“但是管理不代表完全封闭。将来随着发展，生态旅游一定会在国家公园里发展起来。”刘宁表示，相信通过这样的体制建设，会把管理和自然生态需求紧密联结在一起，把责任的边界和保护的价值、开发的潜力紧密协同起来。'''"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "D:\\Design Software\\Anaconda\\envs\\py36\\lib\\site-packages\\smart_open\\smart_open_lib.py:398: UserWarning: This function is deprecated, use smart_open.open instead. See the migration notes for details: https://github.com/RaRe-Technologies/smart_open/blob/master/README.rst#migrating-to-the-new-open-function\n",
      "  'See the migration notes for details: %s' % _MIGRATION_NOTES_URL\n",
      "Building prefix dict from the default dictionary ...\n",
      "Loading model from cache C:\\Users\\Oliver\\AppData\\Local\\Temp\\jieba.cache\n",
      "Loading model cost 0.889 seconds.\n",
      "Prefix dict has been built succesfully.\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "defaultdict(list,\n",
       "            {1: ['刘宁', '时至今日，青海绝不是仅仅保护一两种动物'],\n",
       "             6: ['刘宁', '为了保护藏羚羊不被非法猎取而牺牲生命的索南达杰'],\n",
       "             8: ['刘宁',\n",
       "              '由于生态恢复、植被恢复和降雨量的增加，还有局部气温升高，导致卓乃湖降雨量丰沛，与其他三个湖连起来，一旦漫溢可能冲刷掉下游大量的沼泽、湿地和冻土，甚至对沿途经过的青藏铁路、青藏公路、输电线路等带来影响'],\n",
       "             10: ['刘宁',\n",
       "              '在顺应自然、尊重自然的同时，保护自然绝不是盲目等待自然的演替，而是在人力可能干预的情况下，对于自然灾害可能造成的破坏，进行积极主动的保护'],\n",
       "             11: ['他', '青海和西藏的羌塘、新疆的阿尔金山是联动的'],\n",
       "             13: ['刘宁',\n",
       "              '在青海有一个阿尼玛卿雪山是小冰川时代的遗址，山里有360个湖泊，最高峰是5369米，现在出于保护的原因被关闭了'],\n",
       "             17: ['刘宁',\n",
       "              '相信通过这样的体制建设，会把管理和自然生态需求紧密联结在一起，把责任的边界和保护的价值、开发的潜力紧密协同起来']})"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model_list = load_all_model() #加载模型\n",
    "say_related = get_relatedwords() #获取步骤2保存的搜索结果\n",
    "find_opinion_of_someone(input_news, say_related, model_list)#进行提取"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 步骤6展示"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "如果加载不出展示视频，[请移步](./media)\n",
    "\n",
    "<video src=\"./media/web_show.mp4\" controls=\"controls\" width=\"500\" height=\"300\">您的浏览器不支持播放该视频！</video>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "已经重新整理了项目框架，并把该项目内容归并到一起 ([链接](../ProjectAll/))。目前只更新到言论提取，主页效果如下："
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![home](./img/home1.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "言论提取平台页面："
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![nesextraction](./img/nesextraction1.png)\n",
    "![tableresult](./img/tableresult1.png)\n",
    "![treeresult](./img/treeresult1.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "已经实现了自动提前载入模型，超时时候自动释放模型，以此保证后台服务器不一直占用内存，文本的提取时间大概在0.5s左右。\n",
    "\n",
    "关键词提取，或更多功能有待更新"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 总结\n",
    "\n",
    "1. 模型实现句子向量化对判断两条句子的相似度效果不错，可以根据设定相似度阈值来进行筛选\n",
    "2. 模型的人名提取还存在一定缺陷，有时候提取到的人名过长，比如**隔壁家的老奶奶**，这里实现全部提取，而不是只提取老奶奶。可根据实际情况调整。另外他等指代词替换功能暂时还没有去实现，后续待更新\n",
    "3. 对于一些特殊的句子，模型还暂时提取不出来，可根据具体实际要求，来增加相应的规则。可以直接在代码里面增加，而不用从写。\n",
    "4. 对于一些更特殊的情况，可以考虑在pyltp模型载入时，增加自定义的字典。达到切词，分词等想要的结果。这点可根据实际情况优化\n",
    "5. 由于pyltp等模型加载大概要花费10s时间并且所占据内存过大，故在实际项目中，需要考虑预加载问题和释放问题和并发问题\n",
    "6. 对于web前端框架可以继续优化美观，由于作者不是专业前端人员，该项目前端都是现学现卖的。\n",
    "---"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.9"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
